Skip to content

[fix](be) Apply HAVING before the distinct streaming limit - #70512

Merged
jacktengg merged 1 commit into
apache:masterfrom
jacktengg:DORIS-29952
Nov 16, 2026
Merged

jacktengg merged 1 commit into
apache:masterfrom
jacktengg:DORIS-29952

Conversation

@jacktengg

Copy link
Copy Markdown
Contributor

Background

Distinct streaming aggregation emits new grouping keys as input arrives. BE can use this operator for GROUP BY without aggregate functions. A final aggregate can also contain HAVING and LIMIT.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:
A final distinct streaming aggregate can discard valid groups when it has HAVING and LIMIT. For example, HAVING rejects group 1 and accepts group 2, but LIMIT 1 stops input after group 1. Apply the final limit after HAVING so the query returns group 2.

Details

  • Frequency: Queries can lose rows when an early group fails HAVING and consumes the limit before filtering.
  • Root cause: push() truncated the aggregate block and set _reach_limit before pull() evaluated HAVING.
  • Why existing tests missed it: The existing operator tests covered LIMIT and local preaggregation without a final HAVING filter.
  • Fix: Skip early truncation when conjuncts are present. Reuse reached_limit() after the final HAVING filter. Preserve output accounting for local preaggregation and its handling of rows that bypass deduplication.
  • Other paths and branches: Ordinary aggregation and streaming aggregation already filter before applying the final limit. No release branch is changed.
  • Behavior change: LIMIT counts only groups that pass HAVING in the final distinct streaming aggregate.

Testing

  • Before the fix, all four new BE tests failed. The new SQL suite also failed: it expected 2 and received no row.
  • Built ASAN BE with DISABLE_BE_JAVA_EXTENSIONS=ON DISABLE_BE_CDC_CLIENT=ON ./build.sh --be -j24. The build and its GLIBC_2.17 symbol check passed.
  • Ran ./run-be-ut.sh --run --filter='*DistinctStreamingAgg*' -j24. All 15 tests passed.
  • Ran ./run-regression-test.sh --run -d query_p0/aggregate,query_p0/set_operations,correctness_p0 -s distinct_streaming_having_limit,distinct_streaming_repeated_keys,test_distinct_streaming_agg_local_shuffle. All three suites passed on the local ASAN BE.
  • The new SQL suite uses two rows. It checks one final node for HAVING and LIMIT, then checks the result with distinct streaming on and off at batch sizes 1 and 1024. The expected output was generated by the regression script through the ordinary aggregate.
  • Direct SQL checks returned 2 for both batch sizes. Profiles confirmed the distinct streaming implementation.
  • clang-format v16, check-format, build structure checks, and clang-tidy passed for the changed scope. clang-tidy used the BE and UT compile commands with the compiler wrapper's toolchain paths.

Release note

Fix missing result rows for GROUP BY queries with HAVING and LIMIT when the final aggregate uses distinct streaming aggregation.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Ready

AI Disclosure

  • Tool: OpenAI Codex
  • Generated: Code, tests, comments, and this description
  • Human review: Pending author review

Background

What problem does this PR solve?

Issue Number:
Related PR:
PR Type:

Problem Summary:

Details

Testing

Notes for Reviewers

Release note

Check List (For Author)

  • Behavior changed:
  • Does this need documentation:
  • Pre-submission status:

AI Disclosure

  • Tool:
  • Generated:
  • Human review:

### Background

Distinct streaming aggregation emits new grouping keys as input arrives. BE can use this operator for GROUP BY without aggregate functions. A final aggregate can also contain HAVING and LIMIT.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:
A final distinct streaming aggregate can discard valid groups when it has HAVING and LIMIT. For example, HAVING rejects group 1 and accepts group 2, but LIMIT 1 stops input after group 1. Apply the final limit after HAVING so the query returns group 2.

### Details

- **Frequency:** Queries can lose rows when an early group fails HAVING and consumes the limit before filtering.
- **Root cause:** push() truncated the aggregate block and set _reach_limit before pull() evaluated HAVING.
- **Why existing tests missed it:** The existing operator tests covered LIMIT and local preaggregation without a final HAVING filter.
- **Fix:** Skip early truncation when conjuncts are present. Reuse reached_limit() after the final HAVING filter. Preserve output accounting for local preaggregation and its handling of rows that bypass deduplication.
- **Other paths and branches:** Ordinary aggregation and streaming aggregation already filter before applying the final limit. No release branch is changed.
- **Behavior change:** LIMIT counts only groups that pass HAVING in the final distinct streaming aggregate.

### Testing

- Before the fix, all four new BE tests failed. The new SQL suite also failed: it expected 2 and received no row.
- Built ASAN BE with `DISABLE_BE_JAVA_EXTENSIONS=ON DISABLE_BE_CDC_CLIENT=ON ./build.sh --be -j24`. The build and its GLIBC_2.17 symbol check passed.
- Ran `./run-be-ut.sh --run --filter='*DistinctStreamingAgg*' -j24`. All 15 tests passed.
- Ran `./run-regression-test.sh --run -d query_p0/aggregate,query_p0/set_operations,correctness_p0 -s distinct_streaming_having_limit,distinct_streaming_repeated_keys,test_distinct_streaming_agg_local_shuffle`. All three suites passed on the local ASAN BE.
- The new SQL suite uses two rows. It checks one final node for HAVING and LIMIT, then checks the result with distinct streaming on and off at batch sizes 1 and 1024. The expected output was generated by the regression script through the ordinary aggregate.
- Direct SQL checks returned 2 for both batch sizes. Profiles confirmed the distinct streaming implementation.
- clang-format v16, check-format, build structure checks, and clang-tidy passed for the changed scope. clang-tidy used the BE and UT compile commands with the compiler wrapper's toolchain paths.

### Release note

Fix missing result rows for GROUP BY queries with HAVING and LIMIT when the final aggregate uses distinct streaming aggregation.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Code, tests, comments, and this description
- Human review: Pending author review
@jacktengg

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all four changed files at dc0d2dc and traced FE planning, BE operator selection, task-local execution, buffering, filtering, limits and cleanup. No actionable new findings. No existing inline findings require confirmation. The review converged after 1 round: both complete-scope reviewers and the separate buffered-output risk reviewer returned NO_NEW_VALUABLE_FINDINGS; every risk was independently checked and dismissed with evidence.

Critical checkpoint conclusions:

  1. Goal and correctness: final distinct aggregation now applies HAVING before LIMIT and counts accepted groups once. Rejected groups no longer consume the allowance. The added BE and SQL cases directly target this failure.
  2. Scope and reuse: the production change is focused on push/pull and reuses reached_limit(), consistent with the existing aggregation paths.
  3. Concurrency and locking: changed counters, predicates and buffers are task-local; predicate contexts are cloned and scheduler execution is mutually excluded. No threads, locks, lock order or dependency behavior changed.
  4. Lifecycle and initialization: no new owners or statics. Cached groups drain across child EOF, and existing close clears remaining output/cache/arena after early termination.
  5. Configuration: no new settings or dynamic-configuration behavior; tests use existing switches.
  6. Compatibility: no changed function signatures, protocol or storage formats. No new libc API/dependency is introduced; binary compatibility was not executed or audited.
  7. Thrift/protobuf: no definition changes; cross-branch field-ID checks are inapplicable.
  8. Parallel paths: ordinary aggregation suppresses early limiting with HAVING and filters before final limiting; streaming aggregation also filters before its final limit. Local distinct preaggregation keeps its prior counting/pass-through behavior; final distinct remains outside the spill path.
  9. Conditional checks: the added conjunct guard and final/local split have concrete comments explaining rejected groups and duplicate pass-through rows. No speculative error checks were added.
  10. Test coverage: four BE cases cover rejected-first-group behavior in one block and across child fetches at batch sizes 1 and 1024, including output/count/EOS assertions. SQL compares ordinary and distinct aggregation at both sizes and checks HAVING and LIMIT on the final node. Existing tests cover plain limits, preaggregation/cache, memory fallback and repeated/nullable keys. No material static coverage gap was identified.
  11. Expected results: all four new output values of 2 are correct; RAND() is in [0,1), so key 1 fails and key 2 passes. Ordering and repeated output tags match the suite. Output-generation provenance was reported by the author and was not independently reproduced.
  12. Observability: final output now uses the common processed-row accounting in reached_limit(); existing output/hash/input/memory counters remain available. No additional logging or metrics are needed for this correction.
  13. Transactions and persistence: no transaction, EditLog, failover, version-visibility or delete-bitmap behavior is changed.
  14. Data writes and atomicity: production changes affect query-local aggregation only; no data mutation path changes.
  15. FE/BE variables: no new transmitted state. A retained physical HAVING filter also prevents the translator's separate inner-preaggregation LIMIT pushdown in the regression shape.
  16. Performance: no new hot-loop copies, repeated scans or allocations. Reading beyond rejected groups is required to return the correct result; limit checks and counter updates are constant-time.
  17. Other invariants: nullable conversion still precedes HAVING, predicate failures propagate before counting, and existing memory/COW ownership remains unchanged. The final sweep found no additional actionable issue.

User focus: no additional focus points were supplied; the whole PR was reviewed. No unresolved static candidates or assigned coverage remain. This execution was static inspection only: builds, tests, runtime plans/profiles, style tools and binary checks were not run, and the author's execution claims were not independently reproduced.

Review converged after 1 round(s).

@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@jacktengg

Copy link
Copy Markdown
Contributor Author

run buildall

@jacktengg
jacktengg merged commit e1ed017 into apache:master Nov 16, 2026
35 of 36 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.2.x dev/4.2.x-conflict reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants